<html>
<head><meta charset="utf-8"><title>benchmark suite · t-compiler/wg-polonius · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/index.html">t-compiler/wg-polonius</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html">benchmark suite</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="161043741"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161043741" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161043741">(Mar 18 2019 at 09:56)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> I'm a bit unsure of what to do about the benchmarking task (as well). First of all, are there any benchmarks already that I should use? Second, as I said, would I be free to use Criterion or do you work with some specific benchmarking infrastructure that I should follow? If so, is there anyone I can ask about that?</p>



<a name="161069661"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161069661" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161069661">(Mar 18 2019 at 15:51)</a>:</h4>
<p>I had a meeting with Tobias (my sort of shadow supervisor) today and he suggested I do something similar to what Carter does and generate benchmark input from a few popular crates (or preferably many).</p>



<a name="161156975"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161156975" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161156975">(Mar 19 2019 at 14:47)</a>:</h4>
<p><span class="user-mention" data-user-id="204115">@Albin Stjerna</span> well, yes, we have an existing benchmark suite that the compiler uses, and it has tools for running it. You can view the "official" results on &lt;<a href="https://perf.rust-lang.org/" target="_blank" title="https://perf.rust-lang.org/">https://perf.rust-lang.org/</a>&gt;.  <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> earlier ran this same suite, but with Polonius enabled (<a href="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes" title="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes">discussion here</a>). Repeating this would give you a good "overall picture" of rustc compilation time, though with some caveats, in that the integration right now is not optimal.</p>
<p>However, I think Tobias is correct that it would be good to create more sets of <em>benchmarking facts</em> in polonius. Presently we have only one just set, <a href="https://github.com/rust-lang/polonius/tree/master/inputs/clap-rs" target="_blank" title="https://github.com/rust-lang/polonius/tree/master/inputs/clap-rs">clap-rs</a>, which was a case that was historically problematic. The README for polonius talks about how to generate facts for a random rs file.</p>
<p>Which leaves the question of how to choose the cases we include. Probably we should pick some representatives from the <a href="http://perf.rust-lang.org" target="_blank" title="http://perf.rust-lang.org">perf.rust-lang.org</a> benchmark suite.</p>



<a name="161157236"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157236" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157236">(Mar 19 2019 at 14:50)</a>:</h4>
<p>Looking over that thread, <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> posted some results already we could look at.</p>
<ul>
<li>the optimized algorithm (<a href="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes/near/158969083" title="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes/near/158969083">here</a>)</li>
<li>the location insensitive algorithm (<a href="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes/near/159015362" title="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes/near/159015362">here</a>)</li>
</ul>



<a name="161157445"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157445" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157445">(Mar 19 2019 at 14:52)</a>:</h4>
<p>In terms of benchmark <em>inputs</em>, one challenge is that the polonius inputs are <em>per fn</em>, whereas the perf suite is testing the compilation time of an entire crate.</p>



<a name="161157626"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157626" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157626">(Mar 19 2019 at 14:54)</a>:</h4>
<p>I would not pick tuple-stress (5000x overhead... hmm....) just yet, because I think that's a separate issue. Maybe we can start by picking out some of the smaller test cases that are relatively simple, e.g. keccak.</p>



<a name="161157674"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157674" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157674">(Mar 19 2019 at 14:54)</a>:</h4>
<p>The sources for perf can be found here <a href="https://github.com/rust-lang-nursery/rustc-perf" target="_blank" title="https://github.com/rust-lang-nursery/rustc-perf">https://github.com/rust-lang-nursery/rustc-perf</a> I believe</p>



<a name="161157735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157735">(Mar 19 2019 at 14:55)</a>:</h4>
<p>(<span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> is pretty busy these days but I bet they could help you with running the test suite on your own, if that thread doesn't cover it for you)</p>



<a name="161157779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157779">(Mar 19 2019 at 14:55)</a>:</h4>
<p>I guess we could start by trying to convert keccak into a benchmark just because I <em>know</em> it's one big fn, so that's an easy one. It's also an interesting edge case, iirc. I think it uses a macro to generate a really big control-flow graph.</p>



<a name="161157819"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157819" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157819">(Mar 19 2019 at 14:56)</a>:</h4>
<p>I'm a bit surprised the location insensitive didn't help more there</p>



<a name="161157860"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161157860" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161157860">(Mar 19 2019 at 14:56)</a>:</h4>
<p>(Actually, <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> -- those results -- what exactly were they comparing, do you know? Was that "no NLL at all" vs "NLL with polonius enabled"?)</p>



<a name="161163927"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161163927" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161163927">(Mar 19 2019 at 16:07)</a>:</h4>
<blockquote>
<p>(Actually, <span class="user-mention silent" data-user-id="116266">Santiago Pastorino</span> -- those results -- what exactly were they comparing, do you know? Was that "no NLL at all" vs "NLL with polonius enabled"?)</p>
</blockquote>
<p>which ones specifically?</p>



<a name="161163958"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161163958" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161163958">(Mar 19 2019 at 16:07)</a>:</h4>
<p>was everything against AST if I'm not wrong</p>



<a name="161164107"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161164107" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161164107">(Mar 19 2019 at 16:08)</a>:</h4>
<p>I did do a GitHub search for repositories with code enabling NLL and generated facts from about twelve of them or so, but nothing takes any measurable time to execute on my machine, except fact generation, which for some instances generated hundreds of megabytes of tuples</p>



<a name="161164170"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161164170" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161164170">(Mar 19 2019 at 16:09)</a>:</h4>
<p>(I figured it's not so interesting to analyse code not using NLL, in particular with the hybrid algorithm)</p>



<a name="161168219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161168219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161168219">(Mar 19 2019 at 16:53)</a>:</h4>
<p><span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> ok -- if they were comparing against the ast borrow checker, I guess what we really need to be doing is comparing AST vs NLL vs Polonius.</p>



<a name="161170355"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161170355" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161170355">(Mar 19 2019 at 17:14)</a>:</h4>
<p>yeah, if I remember correct we had</p>



<a name="161170386"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161170386" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161170386">(Mar 19 2019 at 17:14)</a>:</h4>
<p>AST vs NLL and AST vs Polonius</p>



<a name="161173823"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161173823" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161173823">(Mar 19 2019 at 17:46)</a>:</h4>
<p>maybe we can get those numbers into a spreadsheet or something so we can compare better</p>



<a name="161177161"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161177161" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161177161">(Mar 19 2019 at 18:25)</a>:</h4>
<p>Do you mean mine or <span class="user-mention" data-user-id="116266">@Santiago Pastorino</span>'s?</p>



<a name="161182340"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161182340" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161182340">(Mar 19 2019 at 19:27)</a>:</h4>
<blockquote>
<p>to analyse code not using NLL</p>
</blockquote>
<p>Everything in Rust 2018 will be using (some form of) NLL</p>



<a name="161184861"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161184861" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161184861">(Mar 19 2019 at 20:00)</a>:</h4>
<p><span class="user-mention" data-user-id="116155">@Jake Goulding</span> ok, so it means I should look for either edition = 2018 or the NLL feature flag?</p>



<a name="161184990"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161184990" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161184990">(Mar 19 2019 at 20:02)</a>:</h4>
<p>I would think that's more accurate (it's not 100% complete as there's lots of ways to specify the edition)</p>



<a name="161190743"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161190743" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161190743">(Mar 19 2019 at 21:16)</a>:</h4>
<p>Here's a <a href="/user_uploads/4715/ZeEbWfM3acDA99J-3MLY99JT/results.csv" target="_blank" title="results.csv">CSV file</a>  with the results of running Polonius' various algorithms against the crates I found, but as I mentioned they really are mostly at zero.</p>



<a name="161190807"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161190807" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161190807">(Mar 19 2019 at 21:17)</a>:</h4>
<p>Numbers are not per-crate but per-function as ususal</p>



<a name="161190903"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161190903" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161190903">(Mar 19 2019 at 21:18)</a>:</h4>
<p>(if I'm going with something like this, I should try to reverse engineer at least part of what Crater does and do that probably, but this was something of a proof-of-concept of sorts)</p>



<a name="161267848"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161267848" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161267848">(Mar 20 2019 at 17:01)</a>:</h4>
<p><span class="user-mention" data-user-id="116009">@nikomatsakis</span> I generated tuples on keccak and ran them through Polonius, but I still get roughly zero seconds on every function; I'm starting to suspect I'm doing something wrong when generating the inputs</p>



<a name="161267951"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161267951" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161267951">(Mar 20 2019 at 17:02)</a>:</h4>
<p><span class="user-mention" data-user-id="116266">@Santiago Pastorino</span> Could you share how you did to run rustc-perf with Polonius? I haven't managed to figure out how to do that from reading the READMEs in the repository</p>



<a name="161272371"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161272371" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161272371">(Mar 20 2019 at 17:44)</a>:</h4>
<p><span class="user-mention" data-user-id="204115">@Albin Stjerna</span> did you read the topic, there was some discussion about how to do it in there</p>



<a name="161272800"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161272800" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161272800">(Mar 20 2019 at 17:48)</a>:</h4>
<p>Have you tried just running the benchmark suite unedited?</p>



<a name="161272829"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161272829" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161272829">(Mar 20 2019 at 17:49)</a>:</h4>
<p>You might ping <span class="user-mention" data-user-id="116122">@simulacrum</span> -- if they're around, they might be able to help</p>



<a name="161272856"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161272856" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161272856">(Mar 20 2019 at 17:49)</a>:</h4>
<p>It looks like to enable polonius, <a href="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes/near/158930500" title="#narrow/stream/186049-t-compiler.2Fwg-polonius/topic/finding.20perf.20spikes/near/158930500">the idea was to edit the sources and add the command line parameter for now</a></p>



<a name="161278752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161278752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161278752">(Mar 20 2019 at 18:42)</a>:</h4>
<blockquote>
<p>Have you tried just running the benchmark suite unedited?</p>
</blockquote>
<p>I didn't manage to get the build command for collector to produce any binaries, but it compiles now.</p>



<a name="161278803"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161278803" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161278803">(Mar 20 2019 at 18:43)</a>:</h4>
<p>Ok, so the settings enabled when benchmarking are not set in perf/collector but in rustc itself? So I guess I should compile a local copy of rustc and run that</p>



<a name="161279895"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161279895" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161279895">(Mar 20 2019 at 18:54)</a>:</h4>
<p>Ah ok I need Linux to run the performance benchmarks</p>



<a name="161281192"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281192" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281192">(Mar 20 2019 at 19:09)</a>:</h4>
<blockquote>
<p>Ok, so the settings enabled when benchmarking are not set in perf/collector but in rustc itself? So I guess I should compile a local copy of rustc and run that</p>
</blockquote>
<p>wait no</p>



<a name="161281245"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281245" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281245">(Mar 20 2019 at 19:10)</a>:</h4>
<p><span class="user-mention" data-user-id="204115">@Albin Stjerna</span> they are set in the perf/collector -- you can add some custom flags</p>



<a name="161281263"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281263" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281263">(Mar 20 2019 at 19:10)</a>:</h4>
<blockquote>
<p>Ah ok I need Linux to run the performance benchmarks</p>
</blockquote>
<p>oh :)</p>



<a name="161281309"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281309" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281309">(Mar 20 2019 at 19:11)</a>:</h4>
<blockquote>
<blockquote>
<p>Ah ok I need Linux to run the performance benchmarks</p>
</blockquote>
<p>oh :)</p>
</blockquote>
<p>It's fine, I already have a Vagrant box up</p>



<a name="161281320"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281320" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281320">(Mar 20 2019 at 19:11)</a>:</h4>
<p>Just surprised :)</p>



<a name="161281433"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281433" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281433">(Mar 20 2019 at 19:12)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="204115">Albin Stjerna</span> they are set in the perf/collector -- you can add some custom flags</p>
</blockquote>
<p>Ah, right, and I took the link in the Zulip message you linked to a bit too literally, I see</p>



<a name="161281540"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281540" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281540">(Mar 20 2019 at 19:14)</a>:</h4>
<p><span aria-label="wave" class="emoji emoji-1f44b" role="img" title="wave">:wave:</span></p>



<a name="161281541"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281541" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281541">(Mar 20 2019 at 19:14)</a>:</h4>
<p><span class="user-mention" data-user-id="204115">@Albin Stjerna</span> I'll be around for the next ~hour or so if you have questions</p>



<a name="161281597"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281597" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281597">(Mar 20 2019 at 19:14)</a>:</h4>
<p>The reason we need linux is that we only support <code>perf</code> as the stats collection backend</p>



<a name="161281610"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281610" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281610">(Mar 20 2019 at 19:14)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="204115">Albin Stjerna</span> I'll be around for the next ~hour or so if you have questions</p>
</blockquote>
<p>Thanks! I'll let you know. :)</p>



<a name="161281612"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281612" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281612">(Mar 20 2019 at 19:14)</a>:</h4>
<p>I think in the future it's plausible we'll support other things but I don't have the time to do the legwork</p>



<a name="161281660"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161281660" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161281660">(Mar 20 2019 at 19:15)</a>:</h4>
<p>That sounds like an incredibly reasonable set of priorities</p>



<a name="161287722"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161287722" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161287722">(Mar 20 2019 at 20:25)</a>:</h4>
<blockquote>
<p><span class="user-mention silent" data-user-id="116266">Santiago Pastorino</span> Could you share how you did to run rustc-perf with Polonius? I haven't managed to figure out how to do that from reading the READMEs in the repository</p>
</blockquote>
<p>did you figure this out?</p>



<a name="161287734"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161287734" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161287734">(Mar 20 2019 at 20:25)</a>:</h4>
<p>sorry but I have been very busy</p>



<a name="161287753"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161287753" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161287753">(Mar 20 2019 at 20:25)</a>:</h4>
<p>after Rust Latam I'd be a new person <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span></p>



<a name="161289049"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161289049" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161289049">(Mar 20 2019 at 20:40)</a>:</h4>
<blockquote>
<blockquote>
<p><span class="user-mention silent" data-user-id="116266">Santiago Pastorino</span> Could you share how you did to run rustc-perf with Polonius? I haven't managed to figure out how to do that from reading the READMEs in the repository</p>
</blockquote>
<p>did you figure this out?</p>
</blockquote>
<p>I think so, but thanks!</p>



<a name="161289654"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161289654" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161289654">(Mar 20 2019 at 20:47)</a>:</h4>
<blockquote>
<p>after Rust Latam I'd be a new person <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span></p>
</blockquote>
<p>As I usually say to my friends when they are organising things (or otherwise seem overburdened); don't die! (I know it's a very low bar)</p>



<a name="161333533"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161333533" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161333533">(Mar 21 2019 at 10:44)</a>:</h4>
<p>Update: I think the benchmarks are running now. I get no output what so ever, but the fans on my computer sound like it is about to take off from my desk, so something is happening anyway</p>



<a name="161348184"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161348184" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Santiago Pastorino <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161348184">(Mar 21 2019 at 14:08)</a>:</h4>
<blockquote>
<blockquote>
<p>after Rust Latam I'd be a new person <span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span></p>
</blockquote>
<p>As I usually say to my friends when they are organising things (or otherwise seem overburdened); don't die! (I know it's a very low bar)</p>
</blockquote>
<p>hehehe, it's a very time consuming thing to do :)</p>



<a name="161690671"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161690671" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161690671">(Mar 25 2019 at 21:43)</a>:</h4>
<p>Ok, I'm not sure if it was a good idea, but I have now tweaked rustc-perf to also run a Polonius pass, similar to the NLL pass, and it seems to be working: <a href="/user_uploads/4715/KH2FK6bGdDIcXUxpRm7yuLSb/pasted_image.png" target="_blank" title="pasted_image.png">cargo-check wall clock time</a>, <a href="/user_uploads/4715/dENbejnKhCbfITcij3PNk3LW/pasted_image.png" target="_blank" title="pasted_image.png">cargo-check max-rss</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/KH2FK6bGdDIcXUxpRm7yuLSb/pasted_image.png" target="_blank" title="cargo-check wall clock time"><img src="/user_uploads/4715/KH2FK6bGdDIcXUxpRm7yuLSb/pasted_image.png"></a></div><div class="message_inline_image"><a href="/user_uploads/4715/dENbejnKhCbfITcij3PNk3LW/pasted_image.png" target="_blank" title="cargo-check max-rss"><img src="/user_uploads/4715/dENbejnKhCbfITcij3PNk3LW/pasted_image.png"></a></div>



<a name="161690793"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161690793" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161690793">(Mar 25 2019 at 21:44)</a>:</h4>
<p>Tomorrow I'll try running the it for all of them and not just cargo and see how they behave; it would be interesting to do some clustering to figure out if the overhead for using Polonius over NLL is constant or if it only happens in some packages, hopefully I can use that information to extract a few types and construct some benchmarks</p>



<a name="161776472"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161776472" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161776472">(Mar 26 2019 at 16:03)</a>:</h4>
<ul>
<li><a href="/user_uploads/4715/HdeXdS_Vnkfvevcdljzec9qO/pasted_image.png" target="_blank" title="pasted_image.png">Less than 50% performance regression NLL -&gt; Polonius</a><div class="message_inline_image"><a href="/user_uploads/4715/HdeXdS_Vnkfvevcdljzec9qO/pasted_image.png" target="_blank" title="Less than 50% performance regression NLL -&gt; Polonius"><img src="/user_uploads/4715/HdeXdS_Vnkfvevcdljzec9qO/pasted_image.png"></a></div></li>
<li><a href="/user_uploads/4715/J79ZWEeZay1WvOiwTxn-AwbD/pasted_image.png" target="_blank" title="pasted_image.png">More than 50% performance regression NLL -&gt; Polonius</a> <div class="message_inline_image"><a href="/user_uploads/4715/J79ZWEeZay1WvOiwTxn-AwbD/pasted_image.png" target="_blank" title="More than 50% performance regression NLL -&gt; Polonius"><img src="/user_uploads/4715/J79ZWEeZay1WvOiwTxn-AwbD/pasted_image.png"></a></div></li>
<li><a href="/user_uploads/4715/lQ3gOyo_BPPnJFssqDYe0I8u/pasted_image.png" target="_blank" title="pasted_image.png">Bar plot (percentages, log-scale)</a><div class="message_inline_image"><a href="/user_uploads/4715/lQ3gOyo_BPPnJFssqDYe0I8u/pasted_image.png" target="_blank" title="Bar plot (percentages, log-scale)"><img src="/user_uploads/4715/lQ3gOyo_BPPnJFssqDYe0I8u/pasted_image.png"></a></div></li>
</ul>



<a name="161776580"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161776580" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161776580">(Mar 26 2019 at 16:04)</a>:</h4>
<p>I guess a few of those in the middle would make good benchmarking targets?</p>



<a name="161779683"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161779683" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161779683">(Mar 26 2019 at 16:38)</a>:</h4>
<p>Just for kicks, I did some k-means clustering on the same data (that is, relative performance for RSS and wall-clock time), gave it 8 classes (a number I arbitrarily made up), and it came up with one each for html5ever, tuple-stress, wg-grammar, ucd, encoding, inflate, and clap-rs, and everything else it binned in the same category. This seems fairly reasonable to me, as those are outliers and the others have roughly similar performance.</p>



<a name="161782197"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161782197" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161782197">(Mar 26 2019 at 17:05)</a>:</h4>
<p>It's also interesting to note that inflate, ucd, and wg-grammar all have worse relative memory performance than in terms of time, for whatever that might be worth</p>



<a name="161782260"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161782260" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161782260">(Mar 26 2019 at 17:05)</a>:</h4>
<p>I guess that has to have something to do with the number of tuples shoveled between Polonius and rustc?</p>



<a name="161842714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/161842714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#161842714">(Mar 27 2019 at 09:39)</a>:</h4>
<p>Oh, and <a href="/user_uploads/4715/83IR1Dv0V68PZP7wcHTPCITn/pasted_image.png" target="_blank" title="pasted_image.png">this is how the relative difference (in percent) look plotted</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/83IR1Dv0V68PZP7wcHTPCITn/pasted_image.png" target="_blank" title="this is how the relative difference (in percent) look plotted"><img src="/user_uploads/4715/83IR1Dv0V68PZP7wcHTPCITn/pasted_image.png"></a></div>



<a name="162942924"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/162942924" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#162942924">(Apr 09 2019 at 19:35)</a>:</h4>
<p>Ok, so for everyone who wasn't at the meeting but who might be interested (that is, everyone except <span class="user-mention" data-user-id="116009">@nikomatsakis</span>); I hacked together a set of scripts to a) scrape GitHub and <a href="http://crates.io" target="_blank" title="http://crates.io">crates.io</a> for crates, in order of popularity (stars/recent downloads) and b) time <code>cargo check</code> under both NLL and Polonius on them.</p>
<p>I ran each <code>cargo check</code> three times, and recorded the smallest runtime (as measured in wall-clock time). I also used the three records to calculate p-values for the null hypothesis (that is, no difference between NLL and Polonius).</p>
<p><a href="/user_uploads/4715/NyWe7YilTPnKpHlidgUyj1tx/results.csv" target="_blank" title="results.csv">Here is a CSV with about 900 of the results</a></p>



<a name="162943215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/162943215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#162943215">(Apr 09 2019 at 19:38)</a>:</h4>
<p>I also have a list of the crates I discarded because they didn't compile (or, in some cases, because they had wonky pull URLs)</p>



<a name="163364507"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163364507" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163364507">(Apr 15 2019 at 09:15)</a>:</h4>
<p>I tried using Criterion for benchmarking the facts currently in the repo, under both the hybrid method and the optimised Datafrog algorithm, and <a href="/user_uploads/4715/XqzLc3df42MpnUGDRkFMdlwS/pasted_image.png" target="_blank" title="pasted_image.png">the results seem to show a minor improvement for some cases with a minor overhead in others</a>.</p>
<div class="message_inline_image"><a href="/user_uploads/4715/XqzLc3df42MpnUGDRkFMdlwS/pasted_image.png" target="_blank" title="the results seem to show a minor improvement for some cases with a minor overhead in others"><img src="/user_uploads/4715/XqzLc3df42MpnUGDRkFMdlwS/pasted_image.png"></a></div>



<a name="163364587"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163364587" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163364587">(Apr 15 2019 at 09:16)</a>:</h4>
<p>(The clap-rs facts take too long to be viable; Criterion is only good for micro-benchmarks)</p>



<a name="163364591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163364591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163364591">(Apr 15 2019 at 09:16)</a>:</h4>
<p>Unless you want to wait literal hours for results</p>



<a name="163365271"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163365271" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163365271">(Apr 15 2019 at 09:29)</a>:</h4>
<p>Code is <a href="https://github.com/albins/polonius/tree/criterion-%C2%B5bench" target="_blank" title="https://github.com/albins/polonius/tree/criterion-%C2%B5bench">in the polonius-µbench branch of my fork</a> and is executed by just running <code>cargo bench</code> (works on non-nightly as well).</p>



<a name="163720444"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163720444" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163720444">(Apr 19 2019 at 07:53)</a>:</h4>
<p>OK, because I didn't sleep well and had to do something more brainless, I kept running the trial compilation and now I can report from building a little over 3000 crates that there seems to be very little difference between Polonius and NLL: </p>
<ul>
<li>the median p-value shows no significant difference in runtime</li>
<li>geometric means of runtimes across all builds is 0.554 for Polonius and 0.534 for plain NLL.</li>
<li>the vast majority of runtimes falls within the category of between a 0 and 5 % runtime increase when enabling Polonius: <a href="/user_uploads/4715/-lkK49kV9cM67voLTHtT4ocp/Unknown.png" target="_blank" title="Unknown.png">Bar plot of runtime increases in percents</a><div class="message_inline_image"><a href="/user_uploads/4715/-lkK49kV9cM67voLTHtT4ocp/Unknown.png" target="_blank" title="Bar plot of runtime increases in percents"><img src="/user_uploads/4715/-lkK49kV9cM67voLTHtT4ocp/Unknown.png"></a></div></li>
</ul>



<a name="163720454"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163720454" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163720454">(Apr 19 2019 at 07:53)</a>:</h4>
<p>Actually, this is suspiciously good, I'll have to investigate</p>



<a name="163720521"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163720521" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Amanda Stjerna <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163720521">(Apr 19 2019 at 07:55)</a>:</h4>
<p>The most likely explanation is that the entire build process is timed and that dependencies are not built with Polonius enabled; the build process is a bit opaque to me</p>



<a name="163759952"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/186049-t-compiler/wg-polonius/topic/benchmark%20suite/near/163759952" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nikomatsakis <a href="https://rust-lang.github.io/zulip_archive/stream/186049-t-compiler/wg-polonius/topic/benchmark.20suite.html#163759952">(Apr 19 2019 at 19:44)</a>:</h4>
<blockquote>
<p>Actually, this is suspiciously good, I'll have to investigate</p>
</blockquote>
<p>Heh, indeed</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>